Waveband-selective imaging systems and methods

ABSTRACT

An illustrative surgical system may access a plurality of images captured outside a structure within a patient; detect a difference between spectral reflectances of scenes captured in the plurality of images; and identify, based on the detected difference between the spectral reflectances of the scenes captured in the plurality of images, pixels in at least one of the plurality of images that correspond to structure tissue of the structure.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/775,806, filed Jan. 29, 2020, which is a continuation of U.S. patentapplication Ser. No. 15/536,210, filed Jun. 15, 2017, which applicationis the U.S. national phase of International Application No.PCT/US2015/065561, filed Dec. 14, 2015, which claims priority to U.S.Provisional Patent Application 62/092,651, filed Dec. 16, 2014, thecontents of each of which are incorporated herein by reference.

BACKGROUND Field of the Invention

The present invention relates generally to imaging techniques used insurgical procedures, and more particularly to ureter detection andimaging during surgical procedures.

Description of Related Art

A ureter is a narrow tube that carries urine from a kidney to thebladder. Muscles in the ureter walls continually tighten and relaxforcing urine downward, away from the kidneys. About every ten tofifteen seconds, small amounts of urine are emptied into the bladderfrom the ureters.

Injury to ureters is an adverse event associated with surgery involvingthe pelvis or colorectal region. To prevent injury to the ureters duringsurgery, several different techniques have been tried to assist thesurgeon in locating the ureters. For example, the surgeon can call anurologist to thread a small scope with a camera into the urethra toplace a wire into each of the two ureters. Alternatively, lighted stentscan be inserted through the urethra and up through the urinary bladderto access the ureters. However, both of these approaches are disruptiveto the clinical workflow in a majority of benign surgical procedures andan urologist may not be available in some instances.

In another technique to determine the location of the ureters, a smallamount of a radioactive chemical dye (TC99-DTPA) is injected through avein in the patient's arm. The radioactive chemical dye passes throughthe body and is excreted through the urine, and so the radioactivechemical dye passes through the ureters. The ureters are detected by ahand held probe that senses the radioactivity.

In still another technique to locate the ureters, an IV injection, or acatheter-based retrograde injection of a near infrared (NIR) fluorophoreis used to image the ureters using infrared illumination. It wasreported that the ureters could be visualized even when embedded insurrounding tissue, and injury could be assessed in real time usinginvisible light. Eiichi Tanaka, et al. “Real-Time IntraoperativeUreteral Guidance Using Near-Infrared Fluorescence,” J. Urol. 178(5),pgs. 2197-2201 (2007) describe using Indocyanine green (ICG) andCW800-CA, the carboxylic acid form of IRDye™ 800CW NIR dye, from LI-COR(Lincoln, Nebr.) as the NIR fluorophores. Aya Matsui, M. D., et al.,“Real-Time Near-Infrared Fluorescence-Guided Identification of theUreters using Methylene Blue,” Surgery, 148(1) pgs. 78-86 (2010) usemethylene blue as the NIR fluorophore.

Another approach to locate the ureters used infrared thermography.Room-temperature saline was used as an irrigant in the operative fieldso that the whole operative field was cooled temporarily. As theoperative field differentially rewarmed, structures such as bloodvessels rewarmed quickly and appeared as white lines against a darkbackground in an infrared image. A second application of this sameconcept involved filling the upper urinary system with room-temperaturesaline. The pelvis and ureter appeared black against a warmerbackground, which appeared white in an infrared image. See Jeffrey A.Cadeddu, M. D., et al, “Laparoscopic Infrared Imaging,” Journal ofEndourology, Vol. 15, No. 1, pgs. 111-116 (2001).

SUMMARY

Unlike the known techniques used to locate ureters that requireintroduction of a fluorophore, creation of a temperature difference,introduction of an object into the ureters, or introduction of aradioactive dye, selective reflection of light by ureters and by tissuearound the ureters is used to safely and efficiently image the ureters.Thus, endogenous contrast is used to visualize ureters without need forilluminating catheters or the administration, for example, of exogenousfluorophores or radioactive dyes.

In one aspect, a plurality of surgical site scenes is captured. Each ofthe plurality of surgical site scenes is captured from reflected lighthaving a different light spectrum. In one aspect the plurality ofsurgical site scenes is captured about simultaneously, i.e., within thetiming and optical tolerances associated with capturing scenes at thesame time, while in another aspect the plurality of surgical site scenesis captured sequentially.

The plurality of captured surgical site scenes is analyzed to identifyureter tissue in the captured surgical site scenes. A surgical sitescene is displayed on a display device based on the captured surgicalsite scenes. The displayed surgical site scene has ureter tissuehighlighted so that the ureter tissue is easily discerned by thesurgeon.

In one aspect, a surgical site is illuminated with a plurality ofdifferent light spectrums. For example, the surgical site is illuminatedsequentially with at least two of the plurality of different lightspectrums. In one aspect, the plurality of light spectrums includes alight spectrum of wavelengths in a range from 450 nm to 580 nm, a lightspectrum of wavelengths in a range from 640 nm to 750 nm, and a lightspectrum of wavelengths in a range from 900 to 1080 nm.

In one aspect, the analysis of the plurality of captured surgical sitescenes to identify ureter tissue first transforms a location in thesurgical site scene in each of the plurality of captured surgical sitescenes into a ureter signal. The analysis next determines whether theureter signal is indicative of ureter tissue at the location. Forexample, the ureter signal is compared with a threshold. Alternatively,a ratio of the ureter signal and a non-ureter signal is compared with athreshold.

In yet another aspect, the ureter tissue is treated with a dye prior tothe capturing of the surgical site scenes. The dye is different from afluorophore and a radioactive dye, e.g., the dye is Phenzaopyridine.

To carry out these methods, a surgical system includes an image capturesystem. The image capture system is configured to capture a plurality ofsurgical site scenes. Each surgical site scene in the plurality ofsurgical site scenes is captured from reflected light of a differentwaveband in a plurality of wavebands. The surgical system also includesa ureter analysis module coupled to the image capture system. The ureteranalysis module is configured to identify ureter tissue in the capturedsurgical site scenes. The surgical system, in one aspect, also includesan illuminator. The illuminator is configured to illuminate a surgicalsite with each of the plurality of wavebands. A display device of thesurgical system is coupled to the ureter analysis module to receive asurgical site scene and information identifying the ureter tissue inthat scene. The display device is configured to display the surgicalsite scene with a ureter highlighted.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high level diagram of a surgical system that includes aureter analysis module.

FIG. 2 is a graph that illustrates a plurality of light wavebands thatare reflected differently by ureter tissue and non-ureter tissue.

FIG. 3 is a more detailed block diagram of the parts of the surgicalsystem of FIG. 1.

FIG. 4 is a graph showing one way to differentiate between ureter andnon-ureter signals.

In the drawings, the first digit of a reference number indicates thefigure in which the element with that reference number first appeared.

DETAILED DESCRIPTION

Unlike the known techniques used to locate ureters that requireintroduction of a fluorophore, creation of a temperature difference,introduction of an object into the ureters, or introduction of aradioactive dye, a ureter analysis module 135 locates the ureters usinglight reflected by the ureters. The selective reflection of light by theureters and by the tissue around the ureters is used to safely andefficiently image the ureters. Thus, endogenous contrast is used tovisualize ureters without need for illuminating catheters or theadministration, for example, of exogenous fluorophores or radioactivedyes.

FIG. 1 is a high-level diagrammatic view of a surgical system 100, forexample, the da Vinci® Surgical System, including a ureter analysismodule 135. (da Vinci® is a registered trademark of Intuitive Surgical,Inc. of Sunnyvale, Calif.) In this example, a surgeon, using mastercontrols at a surgeon's console 114, remotely manipulates an endoscope112 mounted on a robotic manipulator arm 113 that in turn is mounted oncart 110. The surgeon also remotely manipulates one or more surgicalinstruments coupled to cart 110. There are other parts, cables, etc.associated with the da Vinci® Surgical System, but these are notillustrated in FIG. 1 to avoid detracting from the disclosure.

Further information regarding minimally invasive surgical systems may befound for example in U.S. Pat. No. 9,060,678 (filed Jun. 23, 2007;disclosing Minimally Invasive Surgical System), U.S. Pat. No. 6,837,883B2 (filed Oct. 5, 2001; disclosing Arm Cart for Telerobotic SurgicalSystem), and U.S. Pat. No. 6,331,181 (filed Dec. 28, 2001; disclosingSurgical Robotic Tools, Data Architecture, and Use), all of which areincorporated herein by reference. The use of a teleoperated minimallyinvasive surgical system is illustrative only and is not intended tolimit the invention to this specific system. In view of this disclosure,the aspects described herein can be incorporated in a robotic surgicalsystem or other surgical system that includes the elements necessary toimplement the aspects described herein.

Endoscope 112, in one aspect, is a stereoscopic endoscope, and soincludes two optical channels that pass light from surgical site 103within patient 111, e.g., reflected light and/or fluorescence, to animage capture system 120. Surgical site 103 includes ureters and thetissue commonly found around ureters.

The light reflected by tissue in surgical site 103 including lightreflected by the ureter is captured as a surgical site scene. Eachcaptured surgical site scene includes an image of ureter tissue and animage of non-ureter tissue. Each captured surgical site scene iscontained in a frame of data 121. In one aspect, a plurality of framesis captured, one frame for each of a plurality of wavebands thatilluminate surgical site 103.

For a stereoscopic endoscope, the surgical site scene includes twoscenes, a left scene and a right scene. Two sets of data frames, a leftset and right set, are captured by image capture system 120. The twosets are processed by a ureter detection module 136 of ureter analysismodule 135 to create a left ureter enhanced image and a right ureterenhanced image that are sent to a stereoscopic display unit in surgeon'scontrol console 114. The left ureter enhanced image and the right ureterenhanced image are included in ureter enhanced image 140.

The left ureter enhanced image and the right ureter enhanced image arepresented on the stereoscopic display unit in surgeon's control console114, sometimes referred to as surgeon's console 114 or simply console114, to create a three-dimensional scene of surgical site 103 with theureters highlighted. As noted above, the identification of the uretersis done utilizing the difference between the reflectance of tissuesurrounding the ureters and the reflectance of the ureters. No specialfluorophores, radioactive dyes, temperature differences, or objectedinserted into the ureters are needed. Rather, ureter analysis module 135is configured to use a difference between the spectral reflectance ofureters and the spectral reflectance of tissue surrounding the ureters(non-ureter tissue) to identify and display locations of the ureters insurgical site 103.

FIG. 2 shows spectral reflectance 201 (solid line) of a ureter, i.e., aureter spectral signature 201, as well as spectral reflectance 202(dotted-line) of non-ureter tissue that is in close proximity to uretertissue i.e., a non-ureter spectral signature 202. While visually it isvery hard for a human to detect the color difference between ureterspectral signature 201 and non-ureter spectral signature 202,differences do exist as illustrated in FIG. 2.

Three separate wavebands, e.g., three separate light spectrums, havedistinct reflectance differences between ureter tissue and non-uretertissue. The three separate wavebands are 450 to 580 nanometers (nm) (alow waveband), 640 to 750 nm (a middle waveband), and 900 to 1080 nm (ahigh waveband). Sometimes, these wavebands are referred to as spectrums.

While the low waveband (450 to 580 nm) is in the visible light spectrumand the middle waveband (640 to 750 nm) is partially in the visiblelight spectrum (the visible light spectrum is taken as being between 400nm and 700 nm), reflected visible light in these wavebands does notresult in a color difference that is discernable by the surgeon in thescenes displayed on surgeon's console 114. The color differenceassociated with the reflectance differences is seen more as a slightintensity difference, and small intensity differences are difficult todetect by the human eye in complex lighting and three-dimensional scenestructures, which are typical in surgical site scenes displayed onsurgeon's console 114.

The other part (700 to 750 nm) of the middle waveband (640 to 750 nm)and the high waveband (900 to 1080 nm) are in the near infrared (NIR)portion of the electromagnetic spectrum. Near-infrared light is notdetectable by a human. However, ureter detection module 136 detects thedifferences in spectral reflectance in all three of these wavebands andso can identify the pixels in a scene captured by image capture system120 that correspond to ureter tissue and the pixels in that scene thatcorrespond to non-ureter tissue.

In one aspect, the pixels in a scene that correspond to ureter tissueare highlighted, e.g., false-colored to have a color not typically seenin a surgical site scene, and the resulting scene is displayed onsurgeon's console 114. The highlighted ureters allow the surgeon toeasily discern the locations of the ureters during the surgicalprocedure.

In one aspect, ureter analysis module 135 configures illumination system125 to illuminate the tissue including the ureters with at least twodifferent light spectrums. The light spectrums selected are based ontheir differences in reflectance between ureter tissue and non-uretertissue. Each spectrum is reflected differently by tissue surrounding theureters and by the ureters as illustrated in FIG. 2. In another aspect,filters are used in image capture system 120 to capture framescorresponding to the different light spectrums having differences inreflectance between ureter tissue and non-ureter tissue.

The use of a stereoscopic endoscope and a stereoscopic display areillustrative only and are not intended to be limiting. The aspectsdescribed herein can be applied to systems that do not include thestereoscopic features such as monoscopic endoscopes and/or normaldisplay units.

FIG. 3 is a more detailed illustration of aspects of one example ofsurgical system 100 of FIG. 1. In the embodiment of surgical system 300,an illuminator 310 is coupled to stereoscopic endoscope 301. Illuminator310 includes at least a white light source and optionally may includeone or more infrared illumination sources.

Illuminator 310 is used in conjunction with at least one illuminationchannel in stereoscopic endoscope 301 to illuminate surgical site 303.Alternatively and without loss of generality, illuminator 310 may bereplaced by an illumination source at the distal tip, or near the distaltip, of endoscope 301. Such distal tip illumination may be provided bylight emitting diodes (LEDs), for example, or other illuminationsources.

In one example, illuminator 310 provides white light illumination thatilluminates surgical site 303. In some implementations, illuminator 310can also provide other types of illumination, e.g., non-visible light aswell as a subset of the visible color components that make-up whitelight.

Light from illuminator 310 is directed onto an illumination channel 316that couples illuminator 310 to the illumination channel in endoscope301. The illumination channel in stereoscopic endoscope 301 directs thelight to surgical site 303. The illumination channels can beimplemented, for example, with a fiber optic bundle, a single stiff orflexible rod, or an optical fiber.

In one aspect, each of image capture units 320R, 320L includes an imagecapture sensor 321L, 321R that captures light reflected from surgicalsite 303. Each of image capture sensors 321L, 321R can be multiple CCDsthat each capture a different visible color component; a single CCD withdifferent regions of the CCD that capture a particular visible colorcomponent, etc.; a three-chip CCD senor; a single CMOS image sensor witha color filter array; or a three-CMOS color image sensor assembly, forexample.

Irrespective of the implementation of each of image capture sensors321L, 321R, Each of image capture units 320R, 320L and consequently eachof image capture sensors 321R, 321L captures a plurality of frames ofdata 322L, 322R. In one aspect, the plurality of frames of data includesa frame for each of a plurality of wavebands that illuminate thesurgical site. The frame of data that is captured for each waveband ofthe plurality of wavebands includes a surgical site scene that in turnincludes ureter tissue and non-ureter tissue.

Image capture unit 320L is coupled to a stereoscopic display unit 351 insurgeon's console 350 via a left camera control unit (CCU) 330L. Imagecapture unit 320R is coupled to stereoscopic display unit 351 insurgeon's console 350 via a right camera control unit (CCU) 330R. Cameracontrol units 330L, 330R receive signals from a system process module362 that controls gains, controls capturing images, controlstransferring frames to and from ureter detection module 136, etc. Systemprocess module 362 represents the various controllers including thevision system controllers in system 300. Camera control units 330L, 330Rmay be separate units, or may be combined in a single dual controllerunit. Also, ureter detection module 136 can be implemented in cameracontrol units 330L, 330R.

Display mode select switch 352 provides a signal to a user interface 361that in turn passes the selected display mode to system process module362. Various vision system controllers within system process module 362configure illuminator 310 to produce the desired illumination, configureleft and right camera control units 330L and 330R to acquire the desireddata, and configure any other elements needed to process the acquiredframes so that the surgeon is presented the requested images instereoscopic display unit 351. While in this aspect, the scenesdisplayed on a surgeon's console are discussed, the scenes can also bedisplayed on other monitors located in the operating room or elsewhere.

As shown in FIG. 3, user interface 361, system process module 362, andureter analysis module 135 are grouped as a central controller 360 fordescriptive purposes. Central controller 360 also typically includescolor correction modules that transform the color of the scenes to a newdesired color balance as determined by system process module 362.Optional image processing module 340 receives signals from centralcontroller 360 and processes scenes from the color correction modulesprior to display on stereoscopic display unit 351 in surgeon's console350. The color correction modules and optional image processing module340 are equivalent to modules in known surgical systems and so are notconsidered in further detail.

Below processing associated with a single video stream in a singlechannel is described. However, this processing is applied to the videostreams in both the right and left channels that are provided tostereoscopic display unit 351 in surgeon's console 350. Also,stereoscopic processing equivalent to the prior art stereoscopicprocessing is done with respect to the scenes processed and produced byureter analysis module 135. Since the stereoscopic processing is known,it is not considered in further detail.

In normal operation, i.e., where a color image of a surgical site ispresented on stereoscopic display unit 351, image capture sensor 321Rcaptures a plurality of scenes, one for each color channel in surgicalsystem 300. In another aspect for a ureter analysis mode, appropriateband pass filters are used in image capture sensor 321R so that aplurality of scenes is captured, one for each of the low, medium, andhigh wavebands described above. Each of the plurality of band passfilters passes one of the plurality of wavebands that are reflecteddifferently be ureter tissue and non-ureter tissue. In this aspect ofthe ureter analysis mode, illuminator 310 includes at least a whitelight source and an infrared illumination source.

In another aspect of the ureter analysis mode, surgical site 303 issimultaneously illuminated by a plurality of illumination sources 312,where each of the plurality of illumination sources provides a differentlight spectrum. Each of the different light spectrums is reflecteddifferently by ureter tissue and non-ureter tissue. In this aspect, theimage capture sensor captures a scene for each different light spectrum,i.e., a plurality of scenes are captured about simultaneously, i.e.,within the timing and optical path tolerances associated with thesurgical system with respect to capturing images at the same time.

While in this example a plurality of illumination sources 312 are used,this is illustrative only and is not intended to be limiting.Alternatively, a plurality of filters 318 could be used to filter thelight from light source 311 to provide each of the different wavebandsof light to surgical site 303.

Each of the following configurations are equivalent with respect toproviding frames for analyzing the location of ureter tissue: (1) acamera that has three color filters designed to pass the low, middle,and high wavebands and a broadband light that emits energy over allthree wavebands, (2) a camera that has only one sensor that is sensitiveto all three wavebands and three separate narrowband lights that emitenergy in each of these three wavebands that are turned on sequentiallyand detected by the camera one at a time, and (3) some combination of(1) & (2).

Hence, irrespective of the technique used, in this example, threeframes, i.e., a plurality of frames, are captured. One frame for each ofthe different wavebands that reflect differently from ureter tissue andnon-ureter tissue is captured. Only one of the three captured framescould be analyzed to identify the ureter tissue in the frame. However,in some situations, specular reflections from non-ureter tissue orsurgical instruments and/or depth variations in the tissue may make thesingle frame identification of ureter tissue problematic. Thus, in oneaspect, at least two frames are used to identify the ureter tissue inthe captured surgical site scenes.

Pixels at a same location in the captured surgical site scene in each ofthe frames or in a combination of frames are analyzed to determinewhether the pixels at that location are indicative of ureter tissue. Ifthe pixels at the same location in different frames are indicative ofureter tissue, the location in the surgical site scene is identified asbeing ureter tissue and otherwise is identified as being other thanureter tissue. When all the pixels in the captured frames are analyzed,the locations of ureter tissue in the surgical site scene are known.

In one aspect, the scenes captured in the three frames are combined anddisplayed as a grey scale image, and the locations of ureter tissue arefalse colored, e.g., given a green color, and superimposed with the greyscale image for display on stereoscopic display 351.

Irrespective of how the plurality of frames of light reflected from eachof the plurality of wavebands is obtained, it is necessary to transformthe information in the plurality of frames into a ureter signal and anon-ureter signal at each location in the frames, in one aspect. In thefollowing description, a location in each of the plurality of capturedframes is analyzed to determine whether the pixels at that location inthe plurality of frames represent a ureter signal or a non-uretersignal. This is for purposes of illustration only and is not intended tobe limiting. In practice, a group of locations in a frame or even theentire frame would be processed in one transformation and the uretersignal or non-ureter signal determination made for each location in thegroup or frame.

Also, the locations being referred to are locations in the surgical sitescene as opposed to absolute locations in the captured frames. Forexample, location (x, y) in the surgical site scene may be at location(x, y) in the captured frames for the first and second wavebands and atlocation (x+1, y−1) in the captured frame for the third waveband. Thisdepends on whether the frames captured for the three wavebands areautomatically spatially registered to each other. If the three framesare captured so that the frames are spatially registered to each other,the relationship between locations in the three different frames isfixed and known. An example of an image capture unit in which thecaptured frames are spatially registered to each other is presented inU.S. Pat. No. 8,672,838 B2 (issued Mar. 18, 2014, disclosing “ImageCapture Unit in A Surgical Instrument”), which is incorporated herein byreference.

If the frames are not captured so that the frames are spatiallyregistered to each other, the three captured frames are spatiallyregistered to one another so that the same location in the surgical sitescene is processed from each of the frames. Spatial registration isknown and so is not considered if further detail. See for example,Maintz and Viergever, “An Overview of Medical Image RegistrationMethods,” In Symposium of Medical Image Registration Methods, (1996),which is incorporated herein by reference. See also, Sotiras et al.,“Deformable Medical Image Registration: A Survey,” IEEE Trans MedImaging, 32(7), pp. 1153-1190, (July 2013), which is incorporated hereinby reference.

In one aspect of ureter detection module 136, a waveband to uretersignal transform C is applied to the waveband signals B to obtainureter-based signals D at each location in the surgical site scene. Forexample,

D = C * B $\begin{bmatrix}{ureter} \\{{non} - {ureter}}\end{bmatrix} = {\begin{bmatrix}{{UB}1} & {{UB}2} & {{UB}3} \\{{NB}1} & {{NB}2} & {{NB}3}\end{bmatrix}*\begin{bmatrix}{{Band}1} \\{{Band}2} \\{{Band}3}\end{bmatrix}}$

where

-   -   Band1, Band2, and Band3 are waveband signals representing the        captured surgical site scenes for the low, middle, and high        wavebands, respectively;    -   UB1, UB2, UB3 are a first plurality of weights that convert        plurality of waveband signals Band1, Band2, and Band3 to a        ureter signal ureter; and    -   NB1, NB2, NB3 are a second plurality of weights that convert        plurality of waveband signals Band1, Band2, and Band3 to a        non-ureter signal non-ureter.

After waveband signals Band1, Band2, and Band3 are transformed to aureter signal ureter and to a non-ureter signal non-ureter, the ureterand non-ureter signals are analyzed to determine if it is more likelythat the camera is looking at ureter tissue or non-ureter tissue at eachlocation.

In one aspect, thresholding is performed to determine whether the camerais looking at ureter tissue or non-ureter tissue. For example, anempirical threshold is determined such that when ureter signal ureterhas a predetermined relationship to the threshold, ureter signal ureteris taken as being from ureter tissue. The empirical threshold isdetermined, for example, by illuminating ureter tissue from severaldifferent sources, capturing images of the ureter tissue as describedabove, and performing the transformation just described to determineureter signal ureter for each of the different ureter sources. In oneaspect, the smallest ureter signal ureter_(small) is taken as thethreshold.

In another aspect, the thresholding divides ureter signal ureter bynon-ureter signal non-ureter at each location in the frame, i.e., formsa ratio of the two signals, and compares the ratio with an empiricalthreshold, e.g., determines whether ureter signal ureter is someconstant or multiple greater than non-ureter signal non-ureter. Again,the threshold is determined by using several samples containing uretertissue and non-ureter tissue in the process described above to obtainureter signal ureter and non-ureter signal non-ureter for the samples.These signals are used to determine the threshold.

Thresholding removes noise and false positives from ureter signalureter. Due to noise, ureter signal ureter is quite frequently slightlyabove zero even though the ureter is not present. Using a thresholdallows removing this noise to remove the false positives. Thresholdingalso allows determining the confidence about the ureter detection beforedisplaying a location in the frame as representing ureter tissue to thesurgeon. If a high level of confidence is desired, a high threshold isused. If displaying false positives is acceptable, the threshold is setlower. The magnitude of the threshold determines the level of uretersignal ureter required to label a location in the frame as being uretertissue, and so the threshold really is the level of confidence thesystem has that the ureter is actually at that location in the frame.

In still another aspect, ureter signal ureter vs. non-ureter signalnon-ureter is mapped onto a two-dimensional space, and the region ofthis space that it is highly probably the camera is looking at uretertissue is detected. FIG. 4 is an example of how the two-dimensionalspace is determined for two wavebands. In FIG. 4, the middle wavebandsignals are plotted along the x-axis and the high waveband signals areplotted along the y-axis. For purposes of an example only, the middlewaveband is taken as having a wavelength of 700 nm, and the highwaveband is taken as having a wavelength of about 960 nm. Using FIG. 2for the middle waveband, non-ureter signal non-ureter is scattered aboutdashed vertical line 401, and ureter signal ureter is scattered aboutsolid vertical line 402. For the high waveband, non-ureter signalnon-ureter is scattered about dashed horizontal line 403, and uretersignal ureter is scattered about solid horizontal line 404.

Thus, signals that fall within region 411 are non-ureter signalsnon-ureter, and signals that fall within region 410 are ureter signalsureter. In practice, images of a number of different ureter samples arecaptured using the two actual wavebands, and the non-ureter signalsnon-ureter and the ureter signals ureter obtained using thetransformation described above are plotted. Regions 410 and 411 aredefined, and lines determined that bound the region of ureter signalsureter. Then in a clinical setting, if a ureter signals ureter at agiven location falls within region 410, the location is marked as havingureter tissue, and otherwise the location is considered to havenon-ureter tissue.

After a frame is processed and a determination is made that a region inthe camera's view is the ureter, the frame is displayed to the surgeonwith that region being differentiated from the remainder of the scene.There are again multiple ways to display this information so that theureter is differentiated from the rest of the surgical site scene. Oneway is to include a color overlay over the ureter tissue to alert thesurgeon that the ureter is there. Another way is to change the color ofthe ureter tissue significantly in the displays—the false coloringdescribed above—so that the surgeon sees the ureter tissue as beingdifferent from the surrounding tissue. While yet another way is todisplay warning signs on top of the ureter tissue to signal to thesurgeon the presence of the ureter. In all of these cases, such onoverlay or color changes, in one aspect, can be controlled by thesurgeon, i.e., the surgeon can turn on or off the highlighting of theureters or increase or decrease the color shifts that are applied tovisualize the ureters.

In some aspect, transform C, as described above, is not used, and thethresholding is applied directly in a waveband space using the wavebandsignals to determine whether the location is ureter tissue. In anotheraspect, the ureter signal that comes from applying transform C is useddirectly to display the presence or absence of the ureter.

First plurality of weights UB1, UB2, UB3 and second plurality of weightsNB1, NB2, NB3 in waveband to ureter signal transform C can be determinedmultiple ways. For example, the first and second pluralities of weightscan be determined by first empirically measuring the waveband signalsdirectly reflected by ureter and non-ureter tissue; by modeling thecamera/lighting system and the reflectance of ureter and non-uretertissue; or by some combination of empirical measurement and modeling.Each of these approaches produces a set of waveband signals thatcorrelate to ureter tissue and a set of waveband signals that correlateto non-ureter tissue.

The first and second pluralities of weights are determined combining theset of waveband signals that correlate to ureter tissue and the set ofwaveband signals that correlate to non-ureter tissue and solving for thematrix that minimizes the error. For example,

${DU}{:\begin{bmatrix}1 & 1 & {\ldots} & {1} \\0 & 0 & \ldots & 0\end{bmatrix}}$ ${BU}{:\begin{bmatrix}{U11} & {U12} & \ldots & {U1N} \\{U21} & {U22} & \ldots & {U2N} \\{U31} & {U32} & \ldots & {U3N}\end{bmatrix}}$ ${DN}{:\begin{bmatrix}1 & 1 & {\ldots} & {1} \\0 & 0 & \ldots & 0\end{bmatrix}}$ ${BN}{:\begin{bmatrix}{N11} & {N12} & \ldots & {U1M} \\{N21} & {N22} & \ldots & {U2M} \\{N31} & {N32} & \ldots & {U3M}\end{bmatrix}}$ C = [DUDN] * pinv([BUBN])

where

-   -   DU is a matrix to select the ureter signals;    -   BU is a matrix of the frames captured for each of the three        wavebands for N ureter samples:    -   DN is a matrix to select the non-ureter signals;    -   BN is a matrix of the frames captured for each of the three        wavebands for M non-ureter samples; and    -   pinv is a pseudo inverse.

A pseudo inverse of a matrix is known to those knowledgeable in thefield. A pseudo inverse suitable to use here is referred to as theMoore-Penrose pseudo inverse. A common use of the Moore-Penrose pseudoinverse is to compute a ‘best fit’ least squares solution to the systemof linear equations, which lacks a unique solution. Another use of theMoore-Penrose pseudo inverse is to find the minimum (Euclidean) normsolution to the system of linear equations. In one aspect, the best fitleast squares solution is used.

In the above example, a plurality of different light spectrumsilluminated a surgical site and a frame including a surgical site scenewas captured for each of the different light spectrums, which resultedin a plurality of frames being captured. The plurality of frames wasanalyzed to classify locations in the surgical site scene that wereureter tissue. In another aspect, this same method is used, but thepatient has taken a dye prior to the surgery. The dye is different fromboth a fluorophore and a radioactive dye. A fluorophore absorbs lighthaving an excitation wavelength spectrum and emits light having anemission wavelength spectrum. The process is absorption of incidentlight by the fluorophore and emission of new light by the fluorophore ascontrasted with reflection of the incident light. A radioactive dyegives off radiation that is measured without requiring any incidentlight. The dyes used in this aspect do not emit either new light orradiation, but instead the dyes affect the reflection of the incidentlight.

One example of such a dye is Phenazopyridine, sometimes used in the formof Phenazopyridine Hydochoride. Phenazopyridine is a dye that works as aurinary tract analgesic agent. Phenazopyridine is typically administeredto a patient orally and is rapidly excreted by the kidneys directly intothe urine. Hence, both the urine and the walls of the ureters are dyedby Phenazopyridine, which changes the light absorption properties of theureters, and hence changes at least some of the light reflected by theureters. Thus, in one aspect, Phenazopyridine is administered and thewavebands reflected by ureters tissue treated with Phenazopyridine aredetermined. These wavebands are used in the above analysis.

The various modules described herein can be implemented by softwareexecuting on a processor 131, hardware, firmware, or any combination ofthe three. When the modules are implemented as software executing on aprocessor, the software is stored in a memory 132 as computer readableinstructions and the computer readable instructions are executed onprocessor 131. All or part of the memory can be in a different physicallocation than a processor so long as the processor can be coupled to thememory. Memory refers to a volatile memory, a non-volatile memory, orany combination of the two.

Also, the functions of the various modules, as described herein, may beperformed by one unit, or divided up among different components, each ofwhich may be implemented in turn by any combination of hardware,software that is executed on a processor, and firmware. When divided upamong different components, the components may be centralized in onelocation or distributed across system 300 for distributed processingpurposes. The execution of the various modules results in methods thatperform the processes described above for the various modules andcontroller 360.

Thus, a processor is coupled to a memory containing instructionsexecuted by the processor. This could be accomplished within a computersystem, or alternatively via a connection to another computer via modemsand analog lines, or digital interfaces and a digital carrier line.

Herein, a computer program product comprises a computer readable mediumconfigured to store computer readable code needed for any part of or allof the processes described herein, or in which computer readable codefor any part of or all of those processes is stored. Some examples ofcomputer program products are CD-ROM discs, DVD discs, flash memory, ROMcards, floppy discs, magnetic tapes, computer hard drives, servers on anetwork and signals transmitted over a network representing computerreadable program code. A non-transitory tangible computer programproduct comprises a tangible computer readable medium configured tostore computer readable instructions for any part of or all of theprocesses or in which computer readable instructions for any part of orall of the processes is stored. Non-transitory tangible computer programproducts are CD-ROM discs, DVD discs, flash memory, ROM cards, floppydiscs, magnetic tapes, computer hard drives and other physical storagemediums.

In view of this disclosure, instructions used in any part of or all ofthe processes described herein can be implemented in a wide variety ofcomputer system configurations using an operating system and computerprogramming language of interest to the user.

The above description and the accompanying drawings that illustrateaspects and embodiments of the present inventions should not be taken aslimiting—the claims define the protected inventions. Various mechanical,compositional, structural, electrical, and operational changes may bemade without departing from the spirit and scope of this description andthe claims. In some instances, well-known circuits, structures, andtechniques have not been shown or described in detail to avoid obscuringthe invention.

Further, this description's terminology is not intended to limit theinvention. For example, spatially relative terms—such as “beneath”,“below”, “lower”, “above”, “upper”, “proximal”, “distal”, and thelike—may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positions(i.e., locations) and orientations (i.e., rotational placements) of thedevice in use or operation in addition to the position and orientationshown in the figures.

For example, if the device in the figures is turned over, elementsdescribed as “below” or “beneath” other elements or features would thenbe “above” or “over” the other elements or features. Thus, the exemplaryterm “below” can encompass both positions and orientations of above andbelow. The device may be otherwise oriented (rotated 90 degrees or atother orientations) and the spatially relative descriptors used hereininterpreted accordingly.

Likewise, descriptions of movement along and around various axes includevarious special device positions and orientations. The singular forms“a”, “an”, and “the” are intended to include the plural forms as well,unless the context indicates otherwise. The terms “comprises”,“comprising”, “includes”, and the like specify the presence of statedfeatures, steps, operations, elements, and/or components but do notpreclude the presence or addition of one or more other features, steps,operations, elements, components, and/or groups.

Components described as coupled may be electrically or mechanicallydirectly coupled, or they may be indirectly coupled via one or moreintermediate components. In view of this disclosure, instructions usedin any one of, or any combination of operations described with respectto the augmented display system can be implemented in a wide variety ofcomputer system configurations using an operating system and computerprogramming language of interest to the user.

All examples and illustrative references are non-limiting and should notbe used to limit the claims to specific implementations and embodimentsdescribed herein and their equivalents. The headings are solely forformatting and should not be used to limit the subject matter in anyway, because text under one heading may cross reference or apply to textunder one or more headings. Finally, in view of this disclosure,particular features described in relation to one aspect or embodimentmay be applied to other disclosed aspects or embodiments of theinvention, even though not specifically shown in the drawings ordescribed in the text.

What is claimed is:
 1. A surgical system comprising: a memory storinginstructions; and a processor configured to execute the instructions toaccess a plurality of images captured outside a structure within apatient; detect a difference between spectral reflectances of scenescaptured in the plurality of images; and identify, based on the detecteddifference between the spectral reflectances of the scenes captured inthe plurality of images, pixels in at least one of the plurality ofimages that correspond to structure tissue of the structure.
 2. Thesurgical system of claim 1, wherein the processor is further configuredto execute the instructions to: direct a display device to display aimage included in the plurality of images; and artificially highlightthe pixels that correspond to the structure tissue in the displayedimage.
 3. The surgical system of claim 1, wherein the processor isfurther configured to execute the instructions to direct an illuminatorto illuminate a surgical site with a plurality of different lightspectrums.
 4. The surgical system of claim 3, wherein the directing theilluminator to illuminate the surgical site comprises directing theilluminator to illuminate the surgical site sequentially with at leasttwo of the plurality of different light spectrums.
 5. The surgicalsystem of claim 3, wherein the directing the illuminator to illuminatethe surgical site comprises directing the illuminator to: illuminate thesurgical site with a first light spectrum of wavelengths in a range from450 nm to 580 nm; illuminate the surgical site with a second lightspectrum of wavelengths in a range from 640 nm to 750 nm; and illuminatethe surgical site with a third light spectrum of wavelengths in a rangefrom 900 to 1080 nm.
 6. The surgical system of claim 3, wherein thedirecting the illuminator to illuminate the surgical site comprisesdirecting the illuminator to: illuminate the surgical site with a firstlight spectrum of wavelengths in a range from 450 nm to 580 nm; andilluminate the surgical site with a second light spectrum of wavelengthsilluminating the surgical site with a light spectrum of wavelengths in arange from 640 nm to 750 nm.
 7. The surgical system of claim 1, whereinthe identifying comprises: transforming a location in each of theplurality of captured images into a structure signal; and determiningwhether the structure signal is indicative of structure tissue at thelocation.
 8. The surgical system of claim 7, wherein the determiningcomprises comparing the structure signal with a threshold.
 9. Thesurgical system of claim 7, wherein the determining comprises comparinga ratio of the structure signal and a non-structure signal with athreshold.
 10. The surgical system of claim 1, wherein the identifyingcomprises comparing a structure signal with a threshold.
 11. Thesurgical system of claim 1, wherein the identifying comprises comparinga ratio of a structure signal and a non-structure signal with athreshold.
 12. The surgical system of claim 1, wherein the structuretissue is treated with a dye, the dye being different from a fluorophoreand a radioactive dye.
 13. The surgical system of claim 1, wherein theprocessor is further configured to identify, based on the detecteddifference between the spectral reflectances of the scenes captured inthe plurality of images, pixels in at least one of the plurality ofimages that correspond to non-structure tissue outside the structure.14. The surgical system of claim 13, wherein: the structure is a ureter;the structure tissue is ureter tissue of the ureter; and thenon-structure tissue is non-ureter tissue outside the ureter.
 15. Thesurgical system of claim 1, wherein the plurality of images are capturedwith an endoscope located entirely outside the structure.
 16. A methodcomprising: accessing a plurality of images captured outside a structurewithin a patient; detecting a difference between spectral reflectancesof scenes captured in the plurality of images; and identifying, based onthe detected difference between the spectral reflectances of the scenescaptured in the plurality of images, pixels in at least one of theplurality of images that correspond to structure tissue of thestructure.
 17. The method of claim 16, further comprising: displaying aimage included in the plurality of images on a display device; andartificially highlighting the pixels that correspond to the structuretissue in the displayed image.
 18. The method of claim 16, wherein theidentifying comprises: transforming a location in each of the pluralityof captured images into a structure signal; and determining whether thestructure signal is indicative of structure tissue at the location. 19.The method of claim 16, further comprising identifying, based on thedetected difference between the spectral reflectances of the scenescaptured in the plurality of images, pixels in at least one of theplurality of images that correspond to non-structure tissue outside thestructure.
 20. The method of claim 19, wherein: the structure is aureter; the structure tissue is ureter tissue of the ureter; and thenon-structure tissue is non-ureter tissue outside the ureter.